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ABSTRACT 

An educational program on family resource management 
was evaluated to ijudge the goodness of fit of multiple regression 
equations for predicting rrogram results across multiple sites. The 
applicability and effectiveness of the 11-lesson MONEY SENSE program 
was evaluated in six California locations for 190 clients. Indicators 
of effectiveness were: (1) change in frequency of financial problems; 
(2) change in self-perception of financial status; (3) change in 
average weekly food cost per person; (4) food consumption index 
score; (5) change in percentage of food shopping skills used; and (6) 
change in percentage of money management skills used. The 16 
situational variables posited as influential were concurrently 
entered into two regression equations (restricted and unrestricted) 
for each of the outcome measures, and the validity of the two models 
was asse-,sed. The analysis illustrated the use of a cechnique for 
determining whether the unrestricted equation (one specific to a 
subset) or the restricted aquation (a regression equation for all 
members of a set) represents a better fit. In the example, the fit of 
the unrestricted equation was superior for three outcome measures, 
supporting the intuitive understanding that application of a 
curriculum in multiple locations does not mean that the program 
operates identically in all locations. Six tables provide study data. 
(SLD) 
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Background 



A question of interest in a number of program evaluations is how/whether the outcomes attributed to a 
program are related to characteristics of the program and partid]:)ants, including ethnicity, gender, and 
the nature and extent of their involvement in the program. Increasingly, this question is important not 
juit as a consideration for program improvement, but in concern for compliance with affimriative action. 
For example, are the impacts of a program on non- Anglos equivalent to those exp>erienced by Anglos? Do 
persons who remain in a program for six months learn more than those who leave after three months? Is 
the mix of ingredients (e.g. participants, context, exposure) at one program site equivalent to that of 
another in terms impact on participants? It is this latter question that prompted the analysis 
described in this paj)er. 



The Program Evaluated 

An educational program on family resource management (MONEY SENSE) has been conducted by 
Cooperative Extension staff and trained volunteers in multiple locations in California. Money Sense is 
comprised of 11 lessons, including units on goal setting, budgeting :ord-Keeping, banking, use of credit, 
meal planning and food shopping, savings and inv ^tments, insur< ce, and planning for major purchases. 
Its delivery strategy is to train volunteer Money Sense Advisors, who then "pay back" their training by 
sharing Money Sense content with others. Each Cooperative Extension Home Economist who chooses to 
use Money Sense as part of the county's educational program adapts it to fit the particular situation, 
e.g the potential clientele and volunteers, incentives for participation, cooperating agencies. In the 
three years of its operation prior to this study. Money Sense has been used with many clientele groups, 
including teen-aged foster children participating in Social Services* "Indej)endent Living Program," 
welfare recipients enrolled in Caliromia*s GAIN (Greater Avenues to Independence) workfare program, 
tenants in public housing projects delinquent in rent payments, families served by an Indian HeaUh 
Center, and low-ranked enlisted personnel at a military installation. Each group that has been 
involved with Money Sense has, in effect, been exposed to its own variation of Money Sense. 
Adaptations were made, for example, in the number and focus of lessons used, the Money Sense Advisors 
recruited, the use of small group or one-on-one instruction. 



The Evaluation 



With funding from the state Cooperative Extension system and California's Expanded Food and 
Nutrition Education Program, a statewide impact study of Money Sense was conducted (Wright & 
Varcoe, 1989). The evaluation examined the applicability and effectiveness of Money Sense in the six 
locations where it was used. The number of clients from whom follow-up data were obtained ranged 
from 21 in Humboldt County to 60 at Edwards Air Force Base, for a total of 190. Only the analysis of 
relationships between client/program characteristics and measures of program effectiveness will be 
discussed in this paper. 
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Six indicators of program effectiveness were derived from interviews with Money Sense (MS) clients 

one to three month following their completion of the program: 

Change in frequency of financial problems - Respondents were asked whether they thought 
they had financial problems before they enrolled in Money Sense, and if so, how often (e.g. 
weekly, monthly, occasionally). The question wa repeated for now, after Money Sense. 
Change in self-perception of financial status - Interviewers asked MS clients what they 
considered their financial status to be before MS and currently. Resix)nses ranged from "poor" to 
"excellent." 

Change in average weekly food cost per person - For families of limited means, one area in 
which it is possible to reduce spending is for food. Respondents were asked how much, on the 
average, they spent for food eaten at home before the MS program, and how many persons that 
fed. Then they were asked how much they spend now. Per person weekly food costs for pre-MS 
and now were calculated by dividing the amount specified by the number of persons. The now 
cost was subtracted from the before-MS cost to yield the amount of change. 
Food Consumption Index score - Spending less nwney for food is not good resource management if 
the family*s food consumption patterns do not reflect good dietary practices. To determine 
whether MS clients had adopted desirable dietary practices, a food consumption index was 
calculated for each client. Persons were asked whether, compared to before they started Money 
Sense, they now eat more or less of 15 different foods (e.g. milk/cheese, vegetables, candy, 
potato & tortilla chips). 

Change in Percentage of Food Shopping Skills Used - Interviewers read a list of food shopping 
skills (e.g. do you make a list of the foods you need to buy before going to the store?). After each, 
respondents were asked if they do it now after MS, and if they did it before MS. 
Change in Percentage of Money Management Sk ills Used - Similarly, percentages of applicable 
skills used now and before-MS were calculated for a list of money management skills. Change 
was represented by the difference between the two percentages. 

Sixteen variables available from follow-up data were posited as potentially influential on the extent 
of change among Money Sense clients: 

Time in the Area (TIMEAREA): The length of time an individual/family has lived in an area 
may affect the resources available and accessible to enhance family resource management. 
Marital Status (MARITAL): It was assumed that married persons have more resources to 
contribute to family resource management thaA single persons, and that p>ersons who are 
divorced/separated/widowed have less stable resources to contribute than single persons. 
Number of Money Sense Lessons Completed (NOLESSNS): Presumably the more exposure the 
client has had to Money Sense, the more positive outcomes can be expected. 
Whether the Client Devel oped a Budget as Part of Money Sense (MSBUDCET): One 
interviewer noted that persons who actually wrote out a spending plan as part of the program 
seemed to have nuide more changes in the number of skills used than those who talked about but 
did not prepare a budget. 

Employment Status (EMPLOY): Employment was cc isidered to be a sc»urce of family/ individual 
stability, with full-time employment better than part-time. 

Number of Earners (NOEARNRS): More conttibutors to household income were considered 
better than less. 

Monthly Income pgr Housghnld Member (MOINC): The higher end of the respondent's income 
bracket was divided by the number of persons in the household to provide a rough estimate of 
per capita income. 

Number of Persons in the Household (NOWNUM): The number of persons fed in the household 
now was the indicator of household size. This was considered to be important apart from its 
effect on per capita incor^.** l)ecause of the extra management responsibilities involved v/ith 
increasing household size. 
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Average Weekly Food Costs Prior to Money Sense CPREFOOD) and InCfimt (INCOME): 
These variables were included in the equation because it was believed that they might have an 
effect independent of their use in the calculation of weekly per p)erson food costs and mon^^'y 
income per household member respectively, 

AS£ (AGE): It was assumed that ihe effect of age on family resource management is linear (le,, 

younger persons are less experienced than older ones in managing income and household). This, 

of course, may not be an appropriate assumption, since some older persons may find their 

capacity to manage a shrinking income is diminished. 

Percentage of Food Shoppi ng Skills Used Before Money Sense (PSHNDX): 

Percentage of Money Management Skills Used Before Money Sense (PSKNDX): 

Perceived Financial Status Prior to Money Sense (PRESTAT): 

Frequency of Money Problems Prior to Money Sense (PREPROB): 

Weekly Per P erson Food Shopping Costs Before Money Sense (FWKFDCST): 

These variables, indicators of the MS client's resource management skill and status prior to 

involvement in the program, can be expected to influence the client's ability and motivation to 

make changes in family resource management. Among other things, a high score on any of the 

pre-MS measures puts a ceiling on the amount of change that can be made. 



The sixteen variables posited as potentially influential on the extent of change among Money Sense 
clients were concurrently entered into two types of regression equations for each of the outcome measures: 



The restricted equations test the joint impact of the sixteen explanatory variables in all locations, by 
entering indei>endent variable scores without attention to site. The unrestricted equations test the 
individual impact of the sixteen explanatory variables in each of the locations by creating indep>endent 
variables that represent each of the 96 (16 x 6) site-variable intersections. 

The validity of the two models was assessed with a technique borrowed from econometrics, the 
Seemingly Unrelated Regression (SUR) test for the accuracy of the restriction (Kmenta, 1986). This test 
produces an F ratio, which compares the variability in the restricted equation to the variability in the 
unrestricted equation. In this case, where the total sums of squares were not equal for both equations, 
the F ratio was calculated by comparing the R squares adjusted for degrees of freedom 



Analysis 



2. 



A restricted equation v/here the influence of each explanatory variable was 
forced to be the same for all locations; and 

An unrestricted equation where the influence if each explanatory variable was 
allowed to vary across locations. 




Q-K 



n-Q 



-F 



K,n-Q+1 



where K is the set of restricted regressors, 

Q is the set of unrestricted regressors, and 
n is the number of observations. 



When the F is significant, one rejects the null hypothesis that the effect of each explanatory variable 
is the same across all sites. Instead of being "seemingly unrelated," the explanatory variables are 
actually unrelated across locations. 



Results 



Tables 1 through 6 show the direction of statistically significant (p < .10) predictors of change for both 
the unrestricted (site specific) and restricted (all sites) regression equations. Also shown are the F 
ratio, R square and adjusted R square for each equation, and the F ratio for the SUR goodness of fit test 
of the additional variables. 

For three of the outcome measures, the fit of the unrestricted (site-spedfic) equation is better than that 
of the restricted equation. Qianges in frequency of money problems (Table 1), in weekly per person food 
costs (Table 2), and in consumption of selected foods (Table 3) are better predicted on a site specific basis 
than across all sites. For the other three change measures-percentage of food shopping skills used, 
percentage of monay management skills used, and p>erceived financial status-the restricted (all sites) 
equation is as good a fit (Tables 4-6). 

TABLE 1. Direction of Significant (P<.10) Predictors in the 
Univstricted (Site Specific)and Restricted (All Sites) Regression Analyses 
for Change in Frequency of Money Problems 



Site^Dedfic All 

Variable Hum. DVC Pla. Stan. Ven. AFB Sites 

TIMEAREA - MEG 

PSHNDX NEG 

PSKNDX - POS 

PRESTAT POS " POS 

PREPROB POS POS POS POS POS 

PREFOOD - POS 

NOLESSNS - - POS 

NOEARNRS - NEG - - NEG 

EMPLOY POS 

AGE POS - NEG 



F Ratio for restricted (all sites) regression equation: 10.67347, sig. .(XXX) 
R square = 57544 Adjusted R square = 52152 

F Rario for unrestricted (site specific) regression equation: 3.98379, sig. .0000 

R square = .87900 Adjusted R square = .65836 

F Ratio for SUR goodness of fit test: 1.85, d.f. 16, 52, sig. .0488 

The frequency of money problems before Money Sem,e (P^'iPROB) is a significant predictor ot change in 
frequency of money problems in four of the locations (i.e. the more frequent the pre-Money ?»ense 
problems, the more change was made), but there are other significant predictors unique to some of the 
counties. For example, for clients of the Domestic Violence Center, a decrease in frequency of money 
problems was positively related to the percentage of money management skills (PSKPvJDX) clients had 
used prior to Money Sense. In Ventura Q)unty, the less time the clients (mostly military) had lived in 
the area (TIMEAREA), the more change they exjperienced in the frequenc>' of money problems. 

Changes in weekly per person food costs were better predicted on a site-specific basis than by the 
restricted equation (Table ?X In four locations the amount of money per person spent for food before 
Money Sense (PWKFDCST) was a significant predictor. The more spent, the greater the change was 
likely to be. In three sites, the lower the percentage of food shopping skills used before Money Sense 
(PSHNDX), the more the change (i.e. savings) in per person food costs after Money Sense. 
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TABLE 2. Direction of Significant (P<.10) Predictors in the 
Unrestricted (Site Specific) and Restricted (All Sites) Regression Analyses 
for Change in Weekly Per Person Food Costs 

Site^>edfic All 

Variable Hunv DVC Pla. Stan, Ven, AFB Sites 



TIME AREA 


POS 


MARITAL 




PSI"vJDX 


NEG 


PWKFDCST 


POS 


NOLESSNS 


POS 


AGE 




MOINC 


NEG 


PSKNDX 




NOWNUM 




INCOME 


POS 


PREFOOD 


NEG 





POS 


POS 






POS 










NEG 






POS 


POS 










POS 


POS 






NEG 




POS 







POS 



NEG 


NEG 


POS 


POS 


POS 






NEG 


NEG 




POS 


POS 


ne:; 






NEG 



F Ratio for restricted (all sites) regression equation: 16.52579, sig. .OCXX) 
R square - £7726 Adjusted R square - .63628 

F Ratio for unrestricted (site specific) regression equation; 8.52073, sig. .(XX)0 

R square = .93953 Adjusted R square = .82927 

F Ratio for SUR goodness of fit test: 3.68, d./. 16, 52, sig. .0002 

The equations predicting change in consumption of selected foods vary significantly among the six 
Kioney Sense sites (Table 3). Two variables are sigruficant in more than two locations; one, percentage 
of food shopping skills used before Money Sense (PSHNDX), is negatively related to change in food 
consumption patterns in three sites. That is, the fewer pre-Money Sense food shopping skills used, the 
more improvement in consumption of selected foods after Money Sense. The number of Money Sense 
lessons completed (NOLESSNS) is a significant predictor of change in food consumption patterns in four 
applications; however, the direction of the relationship differs among these locations. 

TABLE 3. Direction of Significant (P<.10) Predictors in the 
Unrestricted (Site-specific) and Restricted (All Sites) Regression Analyses 
for Change in Consumption of Selected Foods 

Site Specific All 

Variable Hum. DVC Pla. Stan. Ven. AFB Sites 

PV^KFOCST - NEG 

PSHNDX - - - NEG NEG NEG 

PREPROB - NEG 

PREFOOD - POS 

NOW NUM - NEG POS POS 

NOLESSNS POS - POS - NEG POS POS 

NOEARNRS - - POS 

MSBUDGT - POS 

MOINC POS 

MARITAL - POS POS 

EMPLOY - - NEG 

AGE - NEG - - NEG 



F Ratio for restricted (all sites) regression equation: 1.86038, sig. .0302 
R square = .19109 Adjusted R square = .08838 

F Ratio for unrestricted (site specific) regression equation: 2.11465, sig. .0020 

R square - .79407 Adjusted R square = .41856 

F Ratio for SUR goodness of fit test: 2.27, d.f. 16, 52, sig. .0135 
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TABLE 4. Direction of Significant (P<.10) Predictors in the 
Unrestricted (Site-specifi**) and Restricted (All Sites) Regression Analyses 
for Qiange in Percentage of Food Shopping Skills Used 

Site Specific All 

Variable Hunt DVC Pla. Stag Yen, AFB Sites 



PWKFDCST - POS 

PSHNDX - NEC NFG MEG NEC MEG NEG 

PREFOOD — NEG 

NOWNUM - POS POS 

NOLESSNS - POS 

NOEARNRS - POS 

AGE — NEG 



F Ratio for restricted (all sites) regression equation: 13,77865, sig. .(KXX) 
R square - £3632 Adjusted R square - 59014 

F Ratio for unrestricted (site specific) regression equation: 3.18398, sig, ,0(XX) 

R square =: £5062 Adjusted R square 58346 

F Ratio for SUR goodness of fa test: £5, d.f. 16, 53, n.s. 

As indicated in Table 4 the use of the site-specific model does not improve the goodness of fit of the 
regression equation for change in percentage of food shopping skills used. The main predictor across all 
sites is the percentage of food shopping skills used before Money Sense (PSHNDX). The fewer skills 
used before, the greater the gain. 

TABLE 5. Direction of Significant (P<.10) Predictors in *he 
Unrestricted (Site-specific) and Restricted (All Sites) Regression Analyses 
for Qiange in Percentage of Money Management Skills Used 

Site Specific All 

Variable Hum. DVC Pla. Stan. Ven. AFB Sites 

TIMEAREA - NEG 

PSKNDX WEG NEG NEG NEG NEG NEG NEG 

PREPROB POS 

NOLESSNS - - NEG 

MSBUDGET - POS 

MARITAL - - POS - NEG 

EMPLOY - POS 



F Ratio for restricted (all sites) regression equation: 46.36177, sig. .0000 
R square = £5480 Adjusted R square = .83637 

F Ratio for unrestricted (site specific) regression equation: 9.06162, sig. .0000 

R square =: .94188 Adjusted R square =: ^794 

F Ratio for SUR goodness of fit test: £2, d.f. 16, 53, n.s. 

The site-specific equation predicting change in percentage of money management skills is not 
significantly better than the restricted (all sites ) model (Table 5). In all cases, the greater the 
percentage of money management skills used prior to Money Sense, the less change was likely to be 
made-a kind of ceiling effect. 
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TABLE 6. Direction of Significant (P<.10) Predictors in the 
Unrestricted (Site-specific) and Restricted (All Sites) Regression Analyses 
for Change in Perceived Financial Status 



SiteSpedfic All 



Variable 


Hum. 


DVC 


Pla. Stan. 


Vpn. 


AFB 


Sites 


TIMEAREA 




NEG 










PSHNDX 
PSKNDX 


NEG 
NEG 










NEG 


PRESTAT 

PREFOOD 

NOEARNRS 


POS 


NEG 


-- NEG 


NEG 


NEG 


NEG 
POS 


MSBUDGET 


NEG 


POS 










MARITAL 
INCOME 




NEG 
POS 




POS 




POS 


EMPLOY 


NEG 













F Ratio for restricted (all sites) regression equation: 1253105, sig. .(X)00 
R square = .61409 Adjusted R square = .856508 

F Ratio for unrestricted (site specific) regression equatioii: 2.63710, sig. .0001 

R square = ^2506 Adjusted R square = 51 220 

F RaHo for SUR goodness of fit test: 55, d.f. 16, 52, n.s. 

The site specific (unrestricted) equation is not a better fit than the all sites (restricted) equation in 
predicting perceived financial status (Table 6). While there are other statistically significant 
predictors of change in perceived financial status, the major influence is the financial status perceived 
before Money Sense (PRESTAT). The lower the pre-Mon^y Sense estimate, the greater the change in 
{perception of financial status. 

Discussion 

This example has some obvious limitations. The six locations included in the analysis served very 
different clientele populations, both within the location and among locations. The descriptive 
variables employed in the regression are those that were available and that could be made to fit the 
regression framework, not necessarily those that best described the learner population and the learning 
situation. The total number of Money Sense clients from the follow-up data have been collected is only 
190, limiting the degrees of freedom available. The outcome mea.sures are mastery levels, not levels on 
a scale with upper limits which few could attain. Perhaps most important, the example was initiated 
without clear appreciation of the uniqueness of each individual client's learning exf)erience. 

Nevertheless, it does illustrate the use of technique for determining whether an unrestricted (i.e. a 
regression specific to a subset) equation or a restricted (i.e. a regression for all members of a set) equation 
represents a better fit. In this example the fit of the unrestricted regression equation is better than that 
of the restricted equation for three of the outcome measures-change in frequency of money problems, in 
weekly per person food costs, and in consumption of selected foods. This supports the intuitive 
understanding that the application of a specific curriculum in multiple locations does not necessarily 
mean that the same "program" is operating in each location. That is, one expects that the combination 
of clientele, nature and extent of exposure to the curriculum, the teacher, and the conditions surrounding 
the learning experience will vary considerably across locations. 
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In this analysis of Money Sense experience, however, the influence of some explanatory variables across 
all sites was especially pronounced for three of the outcome variables--change in percentage of food 
shopping skills used, in percentage of money management skills used, and in perceived financial status. 
In each case the learner's pre-treatment score on the outcome measure was the single most significant 
predictor of gain. The highe. the pre-Money Sense score, the less the gain, probably an artifact of the 
mastery measure used. 

Use of single restricted equation for all locations can lead to inappropriate ideas as to what consHtutes 
a program, and to inaccurate views regarding the relationships between program inputs and outcomes. 
Use of a single restricted equation for each location does not pennit the cross-location (or other subset) 
analysis that is offered by the SUR technique. 
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